Use temporary folder container context manager for wsclean files #145
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
wsclean is the main / only imager currently supported. As it executes it interacts with a set of fits images throughout deconvolution. Depending on their size, number of output channels etc there can be a rather large I/O burden to the disk supporting the filesystem.
Its been found that when many workflows are running concurrently there can be noticeable strain, and sometimes this can causes glitches to the filesystem, slow reads, freezes etc.
This change uses a
hold_then_move_into
context manager that configures wsclean outputs to be written to some other location, and then moved into place along side the corresponding MS. In practice, on a HPC it is useful to set this to refer to a local disk or a ram disk through appropriate environment variables that are evaluated as the wsclean command is being constructed and executed.